Experimental Study of Bidding Strategies for Scientific Workflows using AWS Spot Instances

Spot instance is an auction based Amazon Elastic Compute Cloud (EC2) instance provided by Amazon Web Service (AWS). It aims to help users to reduce their resource renting cost. The price for spot instances sometimes can be as low as one tenth of the price of the same type on demand instances. However, while gaining significantly cost savings on renting resources, users take risks on running instances without any availability guarantees, i.e. running spot instances can be preempted by Amazon at anytime. Spot instances that get pre-empted are not charged for their last hour and some users utilize that feature to run very short jobs. Different bidding strategies have been proposed to ensure the execution performance of tasks submitted to spot instances. In this paper, we present a full EC2 spot instance simulator that uses real EC2 spot pricing history to emulate the spot instance life cycle and expected charges. We review eight of the most popular bidding strategies in both literature and practice and compare them in terms of cost, deadline miss rate and task’s execution length for scientific workflows. Our evaluation provides users a guidance on how different bidding strategies may impact the execution of scientific workflows.